−1.5
−1.0
−0.5
0.0
0.5
1.0
1.5
x
−1.5
−1.0
−0.5
0.0
0.5
1.0
1.5
y
Carte
sian
coordi
nates
−0.2
0.0
0.2
0
.4
0.6
0.
8
1.0
1.2
r
−2.0
−1.5
−1.0
−0.5
0.0
0.5
1.0
1.5
2.0
µ
Polar
coor
dinate
s
Visible la
y
er
(input pixels)
1st hidden la
y
er
(edges)
2nd hidden la
y
er
(corners and
con
tours)
3rd hidden la
y
er
(ob
ject parts)
CAR
PERSON
ANIMAL
Output
(ob
ject iden
tit
y)
x
1
w
1
⇥
x
2
w
2
⇥
+
Elemen
t set
+
⇥
x
w
Elemen
t set
Logistic
Regression
Logistic
Regression
σ
(
w
T
x
)
σ
2
n
n
AI
Mac
hine learning
Represen
tation learning
Deep learning
Example:
Kno
wledge
bases
Example:
Logistic
regression
Example:
Shallo
w
auto
enco
ders
Example:
MLPs
Input
Hand-
designed
program
Output
Input
Hand-
designed
features
Mapping
from
features
Output
Input
F
eatures
Mapping
from
features
Output
Input
Simplest
features
Mapping
from
features
Output
Most
complex
features
Rule-based
systems
Classic
mac
hine
learning
Represen
tation
learning
Deep
learning
•
•
•
•
n
x
1
,
.
.
.
,
x
n
y
w
1
,
.
.
.
,
w
n
f
(
x
,
w
)
=
x
1
w
1
+
·
·
·
+
x
n
w
n
f
(
x
,
w
)
f
(
x
)
f
(
x
,
w
)
f
([0
,
1]
,
w
)
=
1
f
([1
,
0]
,
w
)
=
1
f
([1
,
1]
,
w
)
=
0
f
([0
,
0]
,
w
)
=
0
1900
1950
1985
2000
2015
Y
ear
10
0
10
1
10
2
10
3
10
4
10
5
10
6
10
7
10
8
10
9
Dataset
size
(n
um
b er
examples,
logarithmic
scale)
Iris
MNIST
Public
SVHN
ImageNet
CIF
AR-10
ImageNet10k
ILSVRC
2014
Sports-1M
Rotated
T
vs
C
T
vs
G
vs
F
Criminals
Canadian
Hansard
WMT
English
→
F
rench
Increasing
dataset
size
o
v
er
time
1950
1985
2000
2015
Y
ear
10
1
10
2
10
3
10
4
Connections
p
er
neuron
(logarithmic
scale)
1
2
3
4
5
6
7
8
9
10
F
ruit
fly
Mouse
Cat
Human
Num
b
er
of
connections
p
er
neuron
o
v
er
time
1950
1985
2000
2015
2056
Y
ear
10
−
2
10
−
1
10
0
10
1
10
2
10
3
10
4
10
5
10
6
10
7
10
8
10
9
10
10
10
11
Num
b
er
of
neurons
(logarithmic
scale)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Sponge
Roundworm
Leech
Ant
Bee
F
rog
Octopus
Human
Increasing
neural
net
w
ork
size
o
v
er
time
201
0
201
1
2012
2013
2014
Yea
r
0.05
0.10
0.15
0.20
0.25
0.30
ILS
VR
C c
lass
ifica
tion
er
ror
rate
De
cre
asin
g
erro
r r
ate
ov
er
tim
e